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ABSTRACT 

Content-driven systemic school reform emerged in 
recent years as a major policy alternative in the United States. 

Major reforms include the establishment of state curriculum 
frameworks, the development of student assessments, and the adoption 
of new textbooks tied to the curriculum frameworks. Because 
successful implementation of content-driven reform depends on the 
linkages between state-level policymaking and school-level policy 
implementation, the central question is how state policies have 
changed the ways in which classroom activities; are organized, and 
managed by teachers. This paper presents findings of a study that 
investigated the multilevel linkages of state education reform to 
instructional practices in mathematics. Four stages of empirical 
analyses were conducted with a combination of the following survey 
data: the 1992 Council of Chief State School Officers (CCSSO) state 
survey, the 1990 and 1992 National Assessment of Educational Progress 
Trial State Assessment (NAEP TSA) eighth-grade school survey, and the 
1990-91 Schools and Staffing Survey (SASS) public school 
administrator survey. The data suggest that current state curriculum 
frameworks need to be aligned with instructional materials as well as 
with student assessments. The findings also imply that policy 
implementation is a dynamic organizational process of interactions 
between policy characteristics and the institutional setting in a 
multilayered school system. Thus, systemic school reform needs to 
formulate desired connections between organizational levels in 
normative, structural, and functional arenas. The findings further 
suggest that the impacts of state curriculum and testing mandates on 
pedagogical practices between 1990 and 1992 were not substantial. 
Lastly, a comparison of educational outcomes in California and 
Minnesota showed that California's content-driven reforms have led to 
more successful instructional change than have Minnesota's 
outcome-based reforms. Four figures and 10 tables are included. 
Appendices contain descriptions of school- and state-level 
predictors* (Contains 35 references.) (LMI) 
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States anci districts have used a variety of measures to influence the 
content of the school curriculum. The most commonly enacted policies rely on 
the regulation of inputs such as courses or credits and minutes of instruction. 
When A Nation at Risk called on states to establish or raise credit requirements 
in the core academic subjects of schooling, credit requirements provided little 
substantive 'jaiidance as to the content or purpose of the courses students should 
take. While evidence shows that graduation requirements did improve students' 
exposure to academic subjects over the decade, those courses nevertheless failed 
to provide students with a high level of academic content and critical thinking 
skills (Massell and Fuhrman, 1994). Thus, states became more active in defining 
the desired content of curriculum during the 1980s, and a recent action of many 
states has been to develop or revise state curriculum frameworks and guides in 
response to nationwide professional standard-setting activities (See NCTM, 
1989). 

While proponents of differing views on curriculum and testing concur on 
the need for all students to aim for common learning goals and to achieve at 
high levels, disagreements exist over how these objectives can be promoted, how 
standards should be defined and their attainment measured, and how learning 
goals should be pursued and evaluated (Porter, 1994). On the one hand, those 
who argue for standards-based school reform view coherent instructional 
guidance at the state level as a necessary starting point for local consistency in 
planning and management (Smith and O'Day, 1990; Wheelock, 1995). On the 
other hand, others argue that, given the nature of knowledge and understanding 
and the diversity of human experiences and paths to learning, it is not plausible 
to decide from places far from schools and classrooms what, how, and when 
ideas ought to be taught and student understanding tested (Clune, 1993; Darling- 
Hammond, 1993). 

In order to weigh those arguments for and against centrally prescribed and 
hierarchically implemented content and performance standards, we need to 
know how content-driven state education policies have really worked to bring 
about desired instructional change. Case studies of curriculum reform find that 
new ideas hardly take root in the practice of teachers.’ A remarkable melange of 
traditional and progrvvssivc approach to instruction has been noted: many 

' Despite some progress v.\ students' course-taking, new approaches to teaching math and science as 
advocated by NCTM have made little headway in teaching practices (Education Week, 1994. 
"Report Links Increased Enrollments in Math, Science to Reforms of 80's," Oct. 12, 11). 
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teachers tend to construct hybrids of particular progressive practices grafted onto 
what they ordinarily did in classrooms (Cuban, 1984a; Cohen, 1990). The gap 
between recommended practices and observed practices may be explained from 
two competing perspectives (Boyd, 1987). One explanation locates failure in the 
bureaucracy, arguing that top-down governance stifles practitioner creativity, 
deskills teachers, and hamstrings local initiative. In fact, a Carnegie Foundation 
for the Advancement of Teaching (1988) survey . found that teachers do not 
believe the majority of the reforms have done much positively for the classroom 
and are troubled by the potential for negative impacts. A competing, bottom-up 
explanation identifies obstacles at the school level, where educators interpret 
reforms through the lens of current practice, exploit vague language to weaken 
change efforts, and modify policy goals to conform with local priorities. New 
ideas are not reinforced in the immediate work environment of students and 
teachers, and there is a natural tendency to substitute schoolwork for the goal of 
intentional learning (Elmore, 1995). 

However, both perspectives fail to simultaneously take into account the 
characteristics of adopted policies and the context of school systems in which 
policies are implemented. Thus, more systematic analyses are needed to show 
how both top-down and bottom-up views can be combined to understand the 
relation between policymaking and implementation in a multi-layered school 
system and identify institutional and organizational conditions for successful 
instructional change. In light of tl;cse concerns, I focus on policy instruments 
with the potential to more directly influence the substance of schooling, that is, 
curriculum frameworks or guidelines, student assessments and textbook 
selection, and examine their linkages both at the state level and school level (See 
Figure 1). Even well-coordinated systemic reform policies are not expected to 
automatically lead to desired instructional change.2 Thus, this study attempts to 
identify institutional and organizational factors that mediate the linkage between 
content-driven policies and instructional practices. Further, the effects of 
content-driven policies on instructional practices in mathematics will be 
investigated through multi-level analyses of the 1990 and 1992 National 
Assessment of Educational Progress Trial State Assessment (NAEP TSA) data. 

2 Each of nested hiereurchical layers in a school sytem has a conditional and contributory relation to 
events and outcomes occurring at adjacent ones (Barr and Dreeben, 1983) 

^ 37 states participated in the 1990 TSA in eighth-grade math; and, for the 1992 TSA , which also 
includes fourth-grade reading and math as well as eighth-grade math for the second time, 41 states 
participated. Hawaii, The District of Columbia and territories which participated in the trial 
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Finally, in order to supplement such quantitative analyses, I will look into 
curriculum reform and instructional change in two states, California and 
Minnesota: these states differ in the conditions and outcomes of school reform as 
well as policymaking and implementation activities. ^ 

Figure 1. Linkages between Content-driven Policies and Instructional Practices 

State-level Policymaking 




Note: 



School-level Policy Implementation 



Horizontal Vertical 

Linkages Linkages 



assessment programs are excluded in this study due to their lack of comparability. In both 1990 and 
1992 NAEP TSA, information was also collected from teachers about some instructioncd materials 
and approaches currently used in math class. 

* The choice of the two states are based primarily on the observations of previous case studies (see 
Marsh and Odden, 1993; Cuban, 1984b; Mazzoni, 1993; Massell and Fuhrman, 1994). 
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Linkages between Content-driven Policies and Instructional Practices 



For many years states and districts have periodically devised curriculum 
frameworks or guidelines that were not equipped with challenging content and 
effective pedagogy. Recently, state curriculum frameworks are undergoing a 
process of change and redefinition as states continue to define their policy- 
making and leadership role in education. With the publication of the National 
Council of Teachers of Mathematics (NCTM) Curriculum and Evaluation 
Standards for School Mathematics (1989), states experienced a new round of 
activity with state curriculum frameworks, and worked on ways to represent the 
Standards in state frameworks. In fact, a Council of Chief State School Officers 
(CCSSO) survey found that math frameworks had been revised or developed, or 
were planned in response to the NCTM standards in 41 states (Blank and 
Dalkilic, 1992). In addition to problems with content and pedagogy, state 
curriculum frameworks also shortchanged instructional practices because the 
frameworks were not strongly linked with other policies, particularly tests and 
textbooks. 

Before the release of A Nation at Risk, many states already had testing 
programs in place as a result of the minimum competency testing movement of 
the late 1970s.® Basic skills tests were designed to insure a minimum level of 
learning. However, research indicates that such tests promote a minimum 
approach to level of instruction where emphasis is placed on the rote 
memorization of discrete facts, on absolute "right" answers, and not on critical 
thinking skills. In addition, the framework or guides that many states used did 
not connect to the actual content of what students learned in the classroom. In 
light of such criticisms, many states in the late 1980s and early 1990s began 
modifying their testing programs and controlling textbook adoption. While 
policymakers began to reevaluate the nature of tests and textbooks and their 
effects on classroom instruction, many of the new curriculum frameworks and 
guides appear to play a formal role in the design and development of student 
assessment programs and textbooks. The 1992 CCSSO survey found that state 

* By 1990 23 states were using tests to evaluate students for promotion to the next grade or graduation 
from high school, 38 used tests to monitor students, school and/ or district performance, and 20 used 
them to identify students in need of remediation. However, a field research found that there are no 
substantial added benefits for students, particularly in whether teachers know more about students 
than before (Wilson and Corbett, 1990). Moreover, teachers perceive statewide tests as placing 
more negative demands on their already overcrowded schedule. 



O 

ERIC 



4 



G 



math assessment is linked to the state framework or learning outcomes in 37 
states. At the same time, the survey fovmd that the math framework is used in 24 
states to select or recommend math textbooks. 

Table 1. School-level Linkage between State Curriculum and Testing 



State Testing Impact on Instruction 



State 




No Impact 


Some Impact Great Impact 


Total 


Curriculum 


No Impact 


143 


(59.1) 


76 (31.8) 


22 (9.1) 


241 


Impact on 


Some Impact 


262 


(41.7) 


1123 (25.0) 


241 (33.3) 


1626 


Instruction 


Great Impact 


57 


(43.8) 


468 (18.7) 


966 (37.5) 


1491 




Total 


462 




1667 


1229 


3558 



Statistic DF Value Prob 



Likelihood Ratio Chi-Square 4 1204.992 0.000 

Phi Coefficient 0.622 

Note: Row percent for the frequency of school principals belonging to each cell appears in 
parenthesis. The phi coefficient conveys the strength of relationship between two categorical 
variables. 

Table 2. School-level Linkage between State Curriculum and New Textbooks 



New Textbook Impact on Instruction 



State 




No Impact 


Some Impact Great Impact 


Total 


Curricultim 


No Impact 


71 


(59.1) 


131 


(31.8) 


39 (9.1) 


241 


Impact on 


Some Impact 


240 


(14.7) 


1060 


(65.1) 


328 (20.2) 


1628 


Instruction 


Great Impact 


135 


(9.1) 


890 


(59.4) 


472 (31.5) 


1497 




Total 


446 




2081 




839 


3366 



Statistic DF Value Prob 



Likelihood Ratio Chi-Square 4 115.294 0.000 

Phi Coefficient 0.193 

Note: Row percent for the frequency of school principals belonging to each cell appears in 
parenthesis. The phi coefficient conveys the strength relationship between two categorical 
variables. 
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While many states have curriculum frameworks, and most of states link 
student assessments to content standards, fewer states connect the curriculum 
frameworks to textbook content. This raises the question as to whether policy 
linkages at the state level, that is, linkages between content-driven policies are 
sustained at the school level and how these linkages function to change 
instruction. To answer those questions, I utilize the 1992 NAEP TSA school 
survey data in which principals were asked to rate the extent of instructional ^ 
change that can be attributed to specific policies. The impact of the state 
curriculum mandate on instruction, as reported by principals, is related to the 
impact of the state testing mandates and textbook adoption on instruction. 
Instructional drange that can be attributed to the state testing mandate is strongly 
related to instructional change attributed to the state curriculum mandate (See 
Table 1). On the other hand, instructional change due to the adoption of new 
textbooks is weakly related to instructional change due to state curriculum 
mandates. This indicates that there is also a tenuous link between the state 
curriculum and textbook adoption at the school level (See Table 2). 

Given these horizontal linkages among content-driven policies, the 
central question is whether and how between-state differences in content-driven 
reform policies lead to variation in instructional change at the school level. In 
order to examine such vertical linkages between state policies and school 
practices, state-level policymaking variables extracted from the 1992 CCSSO state 
survey data are related to school-level policy implementation variables extracted 
from the 1992 NAEP TSA principal survey data. In terms of the impact of the 
state curriculum mandate on instruction, there is little difference between states 
that were revising their curriculum frameworks according to the NCTM 
standards as of 1991 and the states that were not (See Table 3). Consequently, the 
NCTM standards were not put in place at the school level until 1992. The modest 
linkage between the state testing mandate and instructional change indicates that 
statewide testing can be a more powerful policy instrument when it is directly 
linked to the state curriculum (See Table 4). On the other hand, state textbook 
adoption policy is hardly related to the impact of new textbooks on instruction, 
which indicates a tenuous linkage between curricular policies and instructional 
practices in loosely coupled school systems (See Table 5). 
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Table 3. Linkage between State Curriculum Policy and Instructional Change 







State Curriculum Impact on Instruction 








No Impact 


Some Impact Great Impact 


Total 


State 

Math 


No/ 

Developing® 


76 (20.5) 


206 (55.5) 


89 (24.0) 


371 


Curriculiun 


Yes^ 


130 (6.6) 


958 (48.7) 


880 (44.7) 


1968 


Policy 


Revising^ 


35 (3.4) 


464 (45.1) 


531 (51.5) 


1030 




Total 


241 


1628 


1500 


3369 


Statistic 




DF Value Prob 





Likelihood Ratio Chi-Square 4 146.629 0.000 

Phi Coefficient 0.222 



® State has no math cumculxim framework or it is developing one as of 1991. 

^ State has math curriculum guide or framework. 

® Math curriculum guide or framework is being revised with NCTM standards. 

Table 4. Linkage between State Testing Policy and Instructional Change 






State Testing Impact on Instruction 








No Impact Some Impact Great Impact 


Total 


State 


No/ 


142 (33.6) 222 (52.7) 


58 (13.7) 


422 


Math 


Developing® 








Testing 


Indirect^ 


144 (18.6) 441 (57.0) 


188 (24.3) 


773 


Policy 


Direct^ 


175 (8.1) 1008 (46.4) 


989 (45.5) 


2172 




Total 


461 1671 


1235 


3367 



Statistic DF Value Prob 

Likelihood Ratio Chi-Square 4 331.187 0.000 

Phi Coefficient 0 . 318 

® State has no statewide math assessment. 

^ Math curriculum guide or framework defines goals or objectives for instruction, and assessment is 
developed or selected to reflect goals and objectives. 

^ Math curriculum guide or framework defines content topics and skills to be assessed. Use of 
learning outcomes for the development of assessment is also regarded as direct linkage. 
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Table 5. Linkage between State Textbook Policy and Instructional Change 



New Textbook Impact on Instruction 







No Impact 


Some Impact Great Impact 


Total 


State 


No Linkage® 


217 (14.8) 


879 (59.8) 


373 (25.4) 


1469 


Math 

Textbook 


Recommend 

b 


101 (12.9) 


509 (65.0) 


174 (22.1) 


784 


Policy 


Select^ 


129 (11.5) 


698 (62.2) 


294 (26.3) 


1121 




Total 


447 


2086 


841 


3374 



Statistic DF Value Prob 

Likelihood Ratio Chi-Square 4 11.084 0.026 

Phi Coefficient 0.057 

® There is no relation between math curriculum framework and textbook adoption, 
b Math curriculum guide or framework is used to recommend a list of textbook with selection being 
made by local districts. 

^ Math curriculum guide or framework is used to select state approved textbooks. 



Institutional and Organizational Reform Mediators for Instructional Change 

Previous studies of state education reform focused on technical and 
practical aspects of policy implementation, but paid little attention to structural 
and normative factors that mediate policy effects. Case studies of curriculum 
reform find that local adoption is most successful when the curriculum 
frameworks are viewed as user-friendly while maintaining their capacity for 
instructional change, fit with local priorities, and allow for flexible adaptation. 
However, any systematic pattern of policy implementation in content-driven 
reform that is generalizable across the states was not examined. This study 
proposes a contingency model of policy implementation and tests the hypothesis 
that educators' responses to top-down policies, which is the key to successful 
implementation, depend on institutional and organizational contexts. On the 
other hand, previous studies of state education reform did not take into account 
multiple forces that compete to affect instructional practices at different layers of 
the school system. Thus, this study examines other external forces of 
instructional change that may supplement or conflict with state policies. 
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In the context of state education reform, policy implementation is depicted 
as a function of fit between major policy characteristics and corresponding 
contextual factors (See Figure 1). First, the compatibility of policy goals with 
professional values is critical to successful policy implementation. As the priority 
of state education policies has shifted from the equality of educational 
opportunity to academic excellence during the last decade, content-driven reform 
policies are more likely to be implemented in states where educators place 
greater emphasis on academic excellence. Secondly, the distribution of power in 
the state education system that fits with policy mechanisms affects the 
favorableness of policy implementation. As state education departments have 
actively engaged in reform activities such as designing tests, establishing 
curriculum guides, and assisting and monitoring districts, the content-driven 
reform policies are more likely to have an impact on instruction in the states 
where educators view curricular influence as more centralized at the top of the 
state education system. Finally, the implementation of content-driven policies is 
constrained by prevailing school conditions which determine policy costs and 
benefits for implementors. Since instructional change depends on the 
availability of organizational resources and capacities, content-driven reform 
policies are more likely to be implemented when they meet the needs of 
individual schools and raise the average policy benefit-cost ratio. 

Building on the hypothesized relationships between top-down policies 
and bottom-up changes, this study attempts not only to examine several external 
sources of instructional change including state policies, but also to explain 
between-school and interstate variation in policy impact. This research involves 
a hierarchical data structure in the 1992 NAEP TSA 8th grade school data where 
schools are nested within states, and requires an application of the hierarchical 
linear model (see Bryk and Raudenbush, 1992). The following items from the 
principal questiormaire are used to identify external forces which are likely to 
cause instructional changes:^ 

[1] Has the state curriculum mandate changed instruction? (State Curriculum) 

[2] Has the state testing mandate changed instruction? (State Testing) 

[3] Have district/school tests changed instruction? (District /School Test) 

‘ Principals' responses over the 3-scale items are gathered across 3100 schools in 40 states. Values 
are assigned to &ree response categories as follows: 1 for 'not at all,' 2 for 'to some extent,' and 3 for 
'to a great extent.' 
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[4] Has public reporting changed instruction? (Public Reporting) 

[5] Has adoption of new textbooks been an impetus for change? (New Textbooks) 

[6] Have national reports changed instruction? (National Reports) 

The above-listed outcome variables are treated as an interval scale and 
related to institutional and organizational factors at the school and state levels 
(See Appendix for a description of those predictors). At the school-level, each 
principal's reported instructional change due to external forces is a function of 
school-level context variables including social and racial composition of schools 
as follows: 

School instructional change = / (Socioeconomic Status, Percent White, 
Professional Training, Teacher Autonomy, Ability Grouping, Communal 
Climate, Program Activities, Absence of Problems, Urban Location, Rural 
Location) 

The school-level predictors are grand-mean centered so that state-level 
outcome variables represent the state average instructional change adjusted for 
between-sl \te differences in those school characteristics.^ At this point, I extract 
some information on state-level institutional characteristics from the 1990-91 
Schools and Staffing Survey (SASS) data to explain interstate variation in those 
outcome variables. Specifically, I pose the following between-state model where 
the state mean instructional change due to external forces depends on 

g 

professional values and the distribution of power in the state education system. 

State mean instructional change = / (State Influence, Local Influence, Principal 
Influence^, Academic Excellence, Basic Skills, Personal Growth, Work Habits) 



^ Grand-mean centering yields an intercept that can be interpreted as an adjusted mean for each 
group, in this case, state. In this way, any state-level effects of organizational variables on 
instructional change are adjusted away. 

® Principals' view of educational goals and curricular influence are considered as state-level 
constructs for both theoretical and practical reasons. Theoretically, this study assumes an 
institutional unity of the state education system in terms of structural and normative aspects. 
Practically, the absence of school-level linkage between the SASS data and NAEP TSA data does 
not allow for the use of those variables as school-level predictors. 

’ Principals' curricular influence is so highly correlated with teachers' curricular influence at the 
state level that only one of the two variables is left in the equation to avoid the problem of 
collinearity. 
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When variance in outcome variables is partitioned into the school and 
state levels, between-school variance is much larger than between-state variance, 
partly due to measurement error in principals' responses (See the variance table 
in Table 6). Greater school-level variance relative to state-level variance means 
that the outcome variables vary more among schools within each state than 
among the 40 states. It is not surprising to see that state-based outcome variables 
(state curriculum and testing mandates) show larger state-level variance relative 
to school-level variance than district or school-based outcome variables (new 
textbooks and district /school tests) do. Percent of between-state variance for 
public reporting and national reports is smaller than that of state-based variables 
but greater than that of school-based variables. Thus, from such a pattern of 
partitioned variance, we may infer the locus of control for policymaking and the 
mechanism in which those policies or programs are implemented. 

As seen in Table 6, the state average instructional change as perceived by 
principals, varies across the types and levels of policies or programs. The greatest 
impact is reported for the state curriculum mandate, followed by state testing, 
new textbooks, district/school tests, public reporting, and national reports, in this 
order. Thus, whether good or bad, state curriculum and testing policies seem to 
be most instrumental in bringing about instructional change. While national 
reports turned out to be least effective as an external leverage of instructional 
change, it needs to be recognized that the survey item "national reports" does not 
differentiate between professional and governmental efforts. For example, the 
NCTM's report on curriculum and evaluation standards in math originated 
from a consensus among teachers that changes were needed in math teaching, 
and the positive NCTM experience encouraged a set of federally funded efforts 
quite different in their genesis and approach. Most of the more recent national 
reports on standards were developed go vernmen tally, rather than from 
professional efforts.^” 



Thus, the relatively smaller impact of national reports on instructional practices does not 
necessarily indicate that national standards developed by professional associations are weak 
policy tools for instructional change than state-legislated guidelines. 



Table 6. HLM Final Model of Instructional Change due to External Forces 



irn 




Estimated Effects 










State 


State 


District/ 


Public 


New 


National 




Curriculum 


Testing 


School Test 


Reporting 


Textbooks 


Reports 



State-level Effects 



Mean Outcome 


2.275*** 


2.140*** 


2.030*** 


2.017*** 


2.112*** 


1.909*** 


State Influence 


0.234*** 


0.165** 


0.060 


-0.007 


0.023 


-0.062* 


Local Influence 


0.002 


0.032 


0.027 


0.015 


0.014 


-0.034 


Principal Influence 


0.086* 


0.047 


0.062^ 


0.008 


0.040 


-0.026 


Academic Excellence 


0.065 


0.139* 


0.049 


0.135** 


0.051 


0.018 


Basic Skills 


0.003 


-0.084 


-0.072^ 


-0.035 


0.018 


0.005 


Personal Growth 


0.066 


-0.063 


-0.026 


0.015 


0.062 


0.027 


Work Habits 


0.018 


0.027 


0.041 


0.001 


0.072^ 


-0.024 


School-level Effects 














Socioeconomic Status 


-0.079*** 


-0.085** 


-0.151*** 


-0.119*** 


-0.032 


0.134*** 


Percent White 


-0.003 


-0.048 


-0.139* 


-0.174** 


-0.017 


-0.022 


Professional Training 


-0.015 


0.003 


0.024* 


0.030** 


0.006 


0.028** 


Teacher Autonomy 


-0.010 


-0.013 


-0.023^ 


-0.036** 


0.015 


-0.006 


Ability Grouping 


-0.001 


0.036 


0.083** 


0.050* 


0.031 


0.025 


Commtmal Climate 


0.004 


0.006 


0.043** 


0.002 


0.037** 


0.079*** 


Program Activities 


0.073*** 


0.063*** 


0.071*** 


0.066*** 


0.060*** 


0.145*** 


Absence of Problems 


0.007 


-0.009 


-0/J12 


-0.023^ 


-0.016 


-0.042** 


Urban Location 


-0.091** 


-0.004 


0.018 


-0.125** 


-0.018 


0.051 


Rural Location 


0.058* 


0.099** 


0.030 


0.070* 


0.022 


-0.093** 






Variance Table 










Percent Variance Partitioned by Fully Unconditional Model 


School-level 


86.6 


75.2 


94.1 


89.5 


95.2 


93.5 


State-level 


13.4 


14.8 


5.9 


10.5 


4.8 


6.5 






Percent Variance Explained by Final Model 




School-level 


2.4 


20.4 


5.2 


5.0 


1.5 


10.8 


State-level 


62.2 


54.5 


23.8 


45.7 


NA 


10.0 



Note: ^ P<.10, * P<.05, ** P<.01, *** P<.001. 

All predictors are standardized (Mean=0; Standard Deviation=l). 'NA' in variance table indicates 
that variance explained by the final model is too small to be properly calculated. 
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At the state level, perceived instructional change due to external forces is 
related to the distribution of power in the state education system as a whole. The 
greater influence the state education department has on school curriculum, the 
greater impact the state curriculum and testing mandates have on classroom 
instruction. It indicates that the state curriculum and testing mandates rely on 
centralized power distribution. At the same time, the relatively small but 
positive effect of principal's influence on state curriculum implementation also 
indicates the possibility that state curriculum reform is actively incorporated by 
leading principals. By contrast, national reports turn out to have more appeal for 
instructional change in states where the state education department has a 
relatively small curricular influence. It indicates that the increasing role of the 
state education department in content-driven systemic reform may counteract 
national policy initiatives. Additionally, principals' view of educational goals 
also mediates or filters their perceived impact of state policies on instruction. 
The requirement that schools administer state tests and publicly report the test 
results has the greatest impact in states where principals place greater emphasis 
on academic excellence. It indicates that the normative basis of test-based school 
accountability policies is academic excellence. 

At the school level, policy implementation is a function of organizational 
conditions. While the percent of variance explained by the final model at the 
school level is quite small, there are some remarkable patterns of statistically 
significant effects across outcomes variables. The negative effects of school SES 
for curriculum- and testing-related policies indicate that their policy benefits are 
greater for low SES schools than for high SES schools. It also suggests that low 
SES schools are more likely to need curricular guidance and technical support as 
provided by state curriculum and testing policies. In contrast, the opposite effect 
of school SES for national reports indicate that high SES schools are more able to 
benefit from the national reports by translating them into instructional practices. 
On the other hand, the school improvement variable (Program Activities) is a 
significant predictor across the board. Its positive effects for all outcome variables 
indicate that the policy implementation costs are lower for schools that are more 
accepting of and active in implementing schoolwide changes. 

The effects of school racial composition (Percent White) are negatively 
significant for district/school tests and public reporting. It indicates that high 
minority schools tend to be more responsive to test-driven accountability 
policies. At the same time, schools that have well-trained but less autonomous 



O 

ERIC 



13 



15 



faculty tend to report that district/school tests and public reporting have a greater 
impact on instruction. Schools which adopt an ability grouping policy also tend 
to be more responsive to test-driven accountability policies. These patterns 
indicate that the attached implementation costs of test-driven accountability 
policies are lower for tightly coupled, professionalized, and academically 
differentiating schools. On the other hand, schools that have better academic 
ethos and social relations (Communal Climate) tend to benefit more from 
selected sources such as local evaluation, textbook adoption, and national 
reports. Other things being equal, urban schools are less likely to respond to state 
curriculum mandates and public reporting requirements than suburban schools, 
which in turn are less responsive than rural schools. This may be attributed to 
the fact that urban schools have a more heterogeneous student population and 
diverse student needs which make standardized top-down policy directives less 
relevant. 



Impact of Content-driven Reform on Instructional Practices 

One of the basic assumptions underlying current educational research is 
that children actively construct knowledge for themselves through interaction 
with their culture and environment (Resnick, 1987).^^ However, the current 
mathematics curriculum often fails to capitalize on the rich informal 
mathematics knowledge and understanding that children bring to instruction, 
thereby divorcing school mathematics from such familiar activities (Romberg 
and Carpenter, 1986). To help anchor mathematics concepts for students, it is 
important to present mathematics in the "everyday" context and encourage 
students to work together in groups to solve problems. Although recent state 
curriculum in math seems to move in the direction that the NCTM standards 
propose, it remains to be seen whether substantial changes occur at the classroom 
level in response to the newly adopted standards and tests. What students learn 
depends on the ways in which they engage in classroom activities. Their 
dispositions toward mathematics are also shaped by such experiences. 



" The constructivist learning theory that many content-standards are also based on creates a 
paradox for them; if students construct knowledge in highly contextualized ways based on their 
diverse, culturally grounded experiences, teaching must be highly adaptive, and curriculum must 
allow for many starting points and pathways (Darling-Hammond, 1994). 
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Consequently, the goal of developing students' mathematical power requires 
careful attention to pedagogy as well as to curriculum (NCTM, 1991). 

Previous studies found inappropriate pedagogy in U.S. math teaching. In 
the 1986 NAEP math study, neither upper nor lower quartile 13-year-olds 
reported much use of constructivist, student-centered approaches such as small 
group work, projects, and lab activities (Dossey et al., 1988). International 
comparison studies also showed that U.S. math instruction was dominated by 
"tell and show" approaches that encouraged "rote learning" and was more 
textbook oriented than that in other coimtries (McKnight et al., 1987). Recent case 
studies have shown that, in contrast to the computational, direct instruction 
approach used in the United States, Japanese math teachers used "hands-on," 
Socratic teaching methods (Stiglei* and Stevenson, 1991). Further, the observed 
pattern of cognitive level and pacing in American math classrooms showed that 
the teacher as manager of classroom activity carries out functions that are not 
inherently pedagogical or intellectually complex; the teacher often arranges for 
intellectual activity by delegating its enactment to children alone or in groups 
(Stodolsky, 1988). By contrast, Asian classrooms are featured by whole-class 
teaching through which lessons are presented to all students in a well-organized 
fashion (Stevenson and Lee, 1995). 

No single teaching method or learning experience can develop a range of 
mathematical activities as efficiently as a broad-based approach that includes 
individual and group work, some projects and activities accompanied by teacher 
explanation, and practical work with concrete materials in concert with practice 
of important techniques. More importantly, all children need to learn complex 
problem-solving skills which are not only desirable but have become necessary 
for responsible citizenship in a modern society (O'Day and Smith, 1993).^^ 
Student-centered instructional practices with a strong emphasis on higher-order 
thinking skills can be considered positive signs of implementation of many 
recent recommendations for the reform of school mathematics (NCTM, 1991).^^ 



” The ingredients of the direct instruction model are teacher pacing at a low cognitive level, and 
teacher specification of tasks, sequences, and structure. Teaching behaviors that are discouraged 
include use of higher-order, divergent, open-ended questions. 

” The most importemt single message of recent psychological theory and research is that complex 
thinking processes are involved in even the most elementary mental activities (Resnick, 1987). 

For example, for instructional practices in 5-8 mathematics, the NCTM standards call for 
increased attention to the following: 1) actively involving students in exploring, conjecturing, 
analyzing, and applying math in both a mathematical and a real-world context, 2) using 
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In order to know the nature and direction of instructional changes due to 
external forces, we need to look more closely into teachers' instructional 
practices. I selected the information on classroom pedagogy from the NAEP TSA 
teacher questionnaire that was administered to eighth-grade math teachers. 
Teachers were selected if they taught the student the subject in which the student 
was assessed. The following items from the 1990 and 1992 NAEP TSA teacher 
survey data are used to "measure" progressive instruction in math class: 

[1] How much emphasis on reasoning/ analysis? (T031511/T044608)^^ 

[2] How much emphasis on commimicating math ideas? (T031512/T044609) 

[3] How often do students work in small groups? (T031403/T044503) 

[4] How often do students write reports/ do projects? (T031410/T044508) 

[5] How often do students measurement and geometry? (T031404/T044512) 

[6] How often do students use calculators? (T031405/T044505) 

[7] How often do students use computers? (T031406/T044506) 

[8] How often do students write about problem-solving? (NA/T044507) 

[9] How often do students discuss math with other students? (NA/T044509) 

[10] How often do students work real-life math problems? (NA/T044510) 

[11] How often do students make up math problems? (NA/T044511) 

[12] How often assess students with wriHien responses? (NA/T044703) 

[13] How often assess students with projects/portfolios? (NA/T044704) 

Rasch measurement model is used to create a construct of "progressive 
instruction" from the survey data and to equate the two tests of different years 
and subjects (See Wright and Stone, 1979). In order to examining instructional 
change over time, teachers are tested two times, in 1990 when the NCTM 
standards were introduced, and in 1992 when the standards were expected to be 
much in place. There is also some corresponding change in the content of survey 
items on instructional practices: the 1992 test adds more NCTM-based practice 
items (e.g., problem-solving and application skills, and performance-based 

appropriate technology for computation and exploration, 3) using concrete materials, 4) being a 
facilitator of learning, 5) assessing learning as an integral part of instruction. 

The purpose of drawing these samples was not to estimate the attributes of the teacher 
population, but to estimate the number of students whose teachers had various attributes and to 
correlate student characteristics and performance with the characteristics of their teachers 
(Johnson et al., 1994. The NAEP 1992 Technical Report, p. 86). 

Original variable names in the dataset appear in parenthesis: the items in the 1990 data precede 
their counterparts in the 1992 data. 'NA' indicates the absence of matching items in the dataset. 
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assessments) to the 1990 test. However, the two tests conduct parallel assessments 
to provide linkage between 1990 and 1992; ther » are six common items covering 
instructional emphasis on reasoning and communication, use of technology, 
small-group work on projects. The items can be categorized into three major 
categories: purpose, practice, and assessment. BIGSTEPS, Rasch measurement 
program, is used to construct objective measures from the responses of 20319 
teachers to 13 items. 

Table 7. Rasch Measurement of Progressive Instruction: Teacher and Item 
Statistics 



Subject 


Logit 


Standard 


Reliability 




Measure 


Error 





Teachers 



M 


-.53 


.49 


.69 


SD 


.89 


.11 




Items 








M 


.OOa 


.01 


1.00 


SD 


1.11 


.00 





® The item difficulty is centered at zero logit 

The items common to both test forms are used to equate the scale 
constructed from the 1992 data with the measures reported for 1990. As shown in 
Table 7, the results of the co-calibration show a perfect item separation 
(reliability=l). In other words, items are very well separated in terms of the 
difficulty of practicing those instructional practices. On the other hand, teacher 
separation reliability is modest (reliabilty=.69), which confirms the observation 
that American teachers are very similar in terms of their classroom practices 
(Goodlad, 1984). Since NAEP data are inappropriate for teacher-level analyses, 
teachers' measures are matched to their students and aggregated to create school- 
level constructs. Thus, my strategy is to conduct a multi-level analysis of the 
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relations between policies and practices by capturing the relevant properties of 
school-level and state-level variables. First, using a sample of schools from each 
state (2554 schools in 33 states), a school-level linear regression model is 
estimated for each school in each state to predict the association of school 
characteristics with progressive instructional practices as follows (See Appendix 
for a description of predictors); 

Progressive instruction = / (Socioeconomic Status, Percent White, Professional 
Training, Teacher Autonomy, Ability Grouping, Communal Climate, Program 
Activities, Absence of Problems, Urban Location, Rural Location) 

Simultaneously, a state-level regression model is estimated for 33 
common states to predict the association of perceived policy impact with actual 
instructional change. Instructional change reported by principals is related to 
instructional practices reported by teachers at the state level. In order to control 
for past instructional practices at the state level, the 1990 state average measure of 
progressive instruction is included as a predictor. Some may question whether 
instructional change over two years can be meaningfully ascribed to policy 
effects. If the perceived impact of the state curriculum and testing policies on 
instructional change had already occurred before 1990 and much of instructional 
practices were in place by the end of last decade, the 1990 progressive instruction 
would be far' from the "pure" pre- treatment measure that is a prerequisite for an 
appropriate adjustment variable in an analysis of policy effects. Nevertheless, the 
validity of controlling for the 1990 status of instructional practices relies on the 
assumption that principals' perception of instructional change is based more on 
the recent impact of state policies. Specifically, I pose the following between-state 
model; 

State mean progressive instruction = / (90 Math Instruction, State Curriculum, 
State Testing, District/ School Tests, Public Reporting, New Textbooks, National 
Reports) 

Although most states mandated some form of statewide curriculum and 
standardized testing for instructional changes, the effects of implementing such 
content-driven policies on the daily practices of school staff and students have 
not been well documented by empirical research. In addition, it remains to be 
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examined what organizational conditions promote or hamper progressive 
instruction at the school level. On the one hand, under externally prescribed 
standards and increased pressures for outcome-based accountability, teachers may 
teach to the statewide test, increase their dependence on textbooks linked to the 
standards, and discourage student-centered classroom activities. On the other 
hand, teachers may actively incorporate content-driven policies into bottom-up 
instructional change rather than simply adapt themselves to the state 
curriculum and testing mandates. As seen in Table 8, whatever is true, there is 
much greater variation among schools than among states, which indicate very 
little systemic convergence among schools within each state. 

At the school level, school socioeconomic status, professional training, 
teacher autonomy, and program activities are all positively related to progressive 
instruction whereas percent of white students in school and ability grouping 
policy are negatively associated with progressive instruction. Other things being 
equal, urban schools tend to be more active in progressive instruction than 
suburban schools. At the state level, the impact of content-driven state education 
policies on math instruction turned out to be statistically insignificant. It suggests 
that the effects of state curriculum and testing policies on instruction are not yet 
systemic as far as authentic pedagogical, practices are concerned and that their 
longer-term effects remain to be evaluated. On the other hand, the impact of 
new textbooks on progressive instruction is positive, whereas the impact of 
district/school tests on progressive instruction is marginally negative. These 
patterns suggest that there was a positive change in the content of new textbooks 

1 Q 

and that the content of current district and school tests needs to be revised. 
Additionally, the insignificant effect of public reporting on instructional practices 
indicates that current test-based accountability policy is not effective for 
instructional change. 



If teachers rather than principals had been surveyed about instructional change, the relationship 
would be much stronger. 

’* Carnegie's survey (1988) also foimd that teachers occasionally strayed from soimd instructional 
practices in order to get students to pass tests. Under high-stakes conditions, there was a decreased 
reliance on their professional judgment in instructional matters, increased time demands, more staff 
reassigrunents, greater pressure, more paperwork, and heightened concern about liability. 
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Table 8. HLM Final Model of Progressive Instruction in Math Class 





Estimated Effects 








Coefficients 


Standard 

Error 


t-Statistic 


p-Value 


State-level Effects 










Mean Outcome 


-.547 


.029 


-18.766 


.000 


90 Math Instruction 


.153 


.034 


4.535 


.000 


New Textbooks 


.081 


.037 


2.157 


.041 


State Curriculum 


-.043 


.034 


-1.291 


.209 


State Testing 


.089 


.058 


1.538 


.136 


District/School Tests 


-.094 


.047 


-1.994 


.057 


Public Reporting 


.014 


.046 


.292 


.773 


National Reports 


-.009 


.031 


-.288 


.776 


School-level Effects 










Socioeconomic Status 


.080 


.018 


4.593 


.000 


Percent White 


-.064 


.018 


-3.606 


.002 


Professional Training 


.191 


.012 


15.823 


.000 


Teacher Autonomy 


.044 


.014 


3.242 


.004 


Ability Grouping 


-.127 


.029 


-4.352 


.000 


Communal Climate 


.015 


.015 


.990 


.332 


Program Activities 


.036 


.016 


2.339 


.028 


Absence of Problems 


.027 


.016 


1.729 


.096 


Urban Location 


.091 


.091 


2.301 


.030 


Rural Location 


.026 


.026 


.757 


.456 




The Variance Table 








Estimated 


Degrees of 


Chi-Square 


p-Value 




Variance 


Freedom 






state-level 


.019 


25 


125.44 


.000 


school-level 


.417 










Percent variance 


Percent variance 




partitioned by base model 


explained by final model 


state-level 




7.5 


51.3 


school-level 




92.5 


13.3 
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Because of the limitation of the data, some findings need to be interpreted 
cautiously. First of all, this study focuses on an aggregate instructional change at 
the state level, thus ignoring enormous variation that may exist among schools 
and teachers in their policy implementation. More innovative schools that 
actively incorporated the state curriculum and testing mandates into their 
instructional practices may have experienced a greater instructional change. 
Secondly, this study examines instructional change between 1990 and 1992 and 
does not take into account the possibility that some systemwide instructional 
change occurred before 1990. It takes a long time before education policies 
relevant to the enacted curriculum are comprehensive and consistent and have 
the authority and power required to bring about intended change in educational 
practices. New instructional practices may already haVe been in place in some 
states in which content-driven policies was established and widely implemented 
long before the early 1990s (See the case of California in the next section). Thirdly, 
this study assumes a certain level of fit between principals' perception of general 
instructional change and teachers' actual practices in math class, and does not 
consider the possibility that teachers have virtually complete control over their 
pedagogical practices or that principals have inaccurate information on 
instructional change. If there were any systematic gap between principais' and 
teachers' responses across the 40 states, the relation between perceived impact ot 
state policies and actual instructional practices might be spurious. Finally, this 
study relates instructional change only to pedagogical practices, and does not take 
into account change in other key instructional aspects.^’ Despite the statistically 
insignificant impact of state curriculum and testing mandates on progressive 
instruction, they may have advanced or upgraded the content of instruction by 
encouraging teachers to cover more challenging content in their math classes. 



Cases: California and Minnesota 

The linkages of state education reform to instructional practices are more 
evident when we compare two states that differ in their approach to school 
reform, especially the timing of curriculum reform, amount of technical support 



” There are three major school delivery standards: enacted curriculum, effective pedagogy, and 
instructionally embedded resources (Porter, 1993). In this respect, the information from the NAEP is 
limited even though it can be used to construct a general indicator of instructional practices. 
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provided/ and the content of curriculum frameworks. California was one of the 
first states to recognize traditional weakness in curriculum guides. Under Bill 
Honig's leadership the frameworks became not only the foxmdation for textbook 
adoption, but also the intellectual core of the state's student assessment program, 
staff development, accountability, and teacher certification (Massell and 
Fuhrman, 1994). Whereas the older frameworks offered lengthy lists of the facts 
and behavioral objectives and inputs such as the number of hours in class, the 
new frameworks are written in a more literary, narrative style to convey the 
information in a compelling and imderstandable way for teachers as well as for 
district curriculum supervisors (See Table 9). In contrast, Minnesota, with a 
strong tradition of local control and weak state intervention, pursued so-called 
"outcome-based education reform." At least initially, the outcome-based reform 
in Minnesota was distinct from the content-driven reform produced in 
California. The philosophy of reform in Minnesota was to posit goals in broad, 
nondisciplinary-based terms without specific reference to pedagogical details. 
However, Mirmesota found that the outcomes alone were not specific enough to 
guide the development of assessments, and it started to develop new curriculum 
frameworks.^® 



Table 9. Math Curriculum Frameworks of California and Minnesota 





California (1992) 


Minnesota (1991) 


Basic Features 


• Document Title: Mathematics 
Framework for California Public 
Schools, K-12 

• State Mandate 

• Year: 1992 

• No. of pages: 220 


• Document Title: Model Learner 
Outcomes for Mathematics 
Education 

• State Mandate 

• Year: 1991 

• No. of pages: 153 



Since the outcome documents could not be used as the basis for a comprehensive approach to 
outcome-based education or as the starting point for creating assessment tools, the Mirmesota 
Department of Education aj.ipointed a "Learner Outcome Framework Team" to develop a "second- 
generation" of curriculum frameworks (Odden, 1994, p. 158). 
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Major Purposes 
for the 
Framework 
(according to 
state 

supervisors) 


• Provide a vision for math 
educators. 

• Provide criteria for instructional 
material 


• Provide assistance to school 
districts developing K-12 
curriculum. 

•Help educators take a global 
view in building a curriculum and 
show how to teach to the national 
standards 


Sources of 


$30,000 armual State budge 


State funds and Northern States 


Funding 




Power grant 


Development 


• Expert Panel 


• Expert Panel 


Activities 


•Model Documents 


•Model Documents 


(described in 


• Teachers involved in writing 




state 


• Successive drafts reviewed 




frameworks) 


• Pilot Study 

• Wide dissemination 




Content 


Defines 13-14 content standards 


Defines 10 model learner outcomes 


Statement 


by grade block. E.g., Standard 9: 


for grades K-12 with an outline of 




Algebra: in grades 5-8, the math 


math content and skills under each 




curriculum should include 


outcome. E.g., Learner Outcome: 




explorations of algebra concepts 


The student will apply number 




and processes ... 


relationships and functions. 


Methods of 


• Examples /models 




Presenting 


•Activities /tasks 


• Activities / tasks 


Content 


• Diagrams/graphics 


• Diagrams /graphics 


Teaching 


• Constructive/ active lessons 


• Constructive/ active lessons 


Practices 


• Technological applications 


• Technological applications 


Addressed by 


• Assessment strategies 


• Assessment strategies 


Frameworks 


• Interdisciplinary connections 

• Discourse and dialogue 

• Flexible use of time and space 


• Interdisciplinary cormections 




• Using tools and technology 


• Using tools and technology 




• Experiments 


• Experiments 




• Writing and graphing 


• Writing and graphing 
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Equity 

Addressed in 
Frameworks 


•Vision/rationale 

• Vignettes 

• Instructional Strategies 

• Materials Selection 

• Assessment approaches 

• Community involvement 

• Policies and recommendations 


•Vision/rationale 

• Staff development 

• Policies and recommendations 


Linkages of 


• Professional Development 


• Professional Development 


Content with 


(Existing) 


(Recommended) 


State Policies 


• State Assessment 


• State Assessment 




(Recommended) 

• Materials and texts (Existing) 


(Recommended) 

• Technology Integration 
(Recommended) 



Note: The table is based on and reconstructed from the findings of the most recent study of state 
curriculum frameworks in mathematics (See Blank, 1995). 

Such differences between the two states in their policymaking for content- 
driven reform are expected to bring about corresponding differences in 
instructional change at the school-level. As seen in Figure 2, the overall impact 
of state curriculum and testing mandates on instructional practices as perceived 
by principals is greater in California than in Minnesota. Further, the differences 
in reported impact are notable in "great" impact category, which indicates more 
drastic instructional change across the system in California. These differences can 
be understood in terms of quite different reform paths taken by the two states. 
Additionally, the greater impact of new textbooks in California is also attributed 
to the state's policy of linking challenging curriculum to textbook adoption. On 
the other hand, the overall impact of district/school tests and public reporting on 
instructional change is greater in Minnesota than in California although the 
latter exceeds the former in "great" impact category. It indicates that Minnesota' 
outcome-based reforms may have led to relatively widespread but incremental 
instructional change at the school level. 



O 

ERIC 



26 



24 



Figure 2. External Sources of Instructional Change 




State State Testing Dist/Schl Public New National 

Curriculum Tests Reporting Textbooks Reports 



Note: Vertical number represents the percent of principals in each state who report great or some 
impact of horizontal categories on instruction in math classes (see an illustrated bar below) 

— > percent of principals who reporl "great" impact (upper area) 

— > percent of principals who report "some" impact (lower area) 

The two states' different policy initiatives also indicate different barriers at 
the stage of policy implementation. While California used curriculum 
frameworks as vehicles for setting the criteria for the adoption of new textbooks, 
the state faced the difficulty of getting publishers to readily respond within the 
time-frame. In addition, the shorter cycle of textbook adoption poses a burden on 
local districts and teachers. In contrast, Minnesota's outcome-based plan 
anticipates that schools, not school districts, will develop their own curriculum. 
But many local schools and districts lacked the ability to create or assemble their 
own curriculum. Instead, they relied on publishers and other external agents for 
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both curricular materials and staff development activities to support the teaching 
of those curricular. In addition to such differences in the technical aspects of 
policy implementation, the two states also differ in the structural and normative 
conditions of their education systems that were identified as critical by the 
statistical analysis. As seen in Table 10, principals in California do not only place 
greater emphasis on academic excellence as most important educational goals but 
also view the state department of education as most influential actor in 
curricular decisionmaking than do principals in Minnesota. Those two factors 
together explain part of the differences in their reported impact of state policies. 



Table 10. Profile of California and Minnesota: Professional Values and Power 
Distribution 







CA 


MN 




Basic Skills 


77.2 


77.1 


Professional 


Academic Excellence 


51.4 


42.4 


Values^ 


Work Habits 


50.7 


56.6 




Personal Growth 


65.1 


78.9 




State Influence 


75.3 


49.6 


Power 


Local Influence 


50.5 


45.2 


Distribution^ 


Principal Influence 


58.0 


55.3 




Teacher Influence 


59.4 


65.8 



® Numbers represent percentage of public school principals in each state who choose each of these 
goals as one of the three that they considered most important. 

^ Numbers represent percentage of public school principals in each state who report each of these 
groups as having a great deal of influence; if they responded with a 5 or 6 on a 6-point scale of 
influence, with 6 representing a great deal of influence. 

Despite different policymaking and implementation activities in the two 
states, it remains to be seen what teachers are really doing in their classrooms. As 
seen in Figure 3, 8th graders in California tend to have more opportunity to 
learn higher-order skills in math than 8th graders in Minnesota. More 
specifically, teachers in California tend to more frequently have their students 
write about problem-solving, write reports, do projects and make up math 
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problems, and do more performance-oriented assessments. Those differences in 
pedagogical practices can be explained partly by differences in the role of the state 
curriculum frameworks for instructional guidance. In addressing pedagogical 
practices in the curriculum frameworks, California used contextualized examples 
and vignettes illustrating teaching practices in relation to specific content and 
classroom situations, whereas Minnesota used lists and outlines arraying 
teaching practices with little or no detail about how practice is to be used within 
classrooms or by teachers (See Table 9). At the same time, some of the differences 
in progressive instruction can be explained by differences in their school-level 
organizational conditions. As seen in Figure 4, California has relatively more 
favorable school organizational conditions for bottom-up change than 
Minnesota: California excels Minnesota in the school-level capacity-building and 
improvement such as professional training, communal climate, and program 
activities. 
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Figure 3- Progressive Instructional Practices in Math 




Write about 


Write 


Discuss 


Work real- 


Makeup 


Assess 


Assess 


problem- 


reports/do 


math 


life 


math 


w /written 


w /projects 


solving 


projects 


w/others 


problems 


problems 


responses 





Note: Vertical number represents the percent of students in each state whose teachers report 
regular classroom practices for horizontal categories in their math classes (see an illustrated bar 
below) 



— > percent of students whose teachers do the practices "daily or weekly" 
("weekly or monthly" for assessment items) 

— > percent of students whose teachers do the practices "once or twice a month" 
("once or twice a year" for non-assessment items) 
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BEST COPY AVAIUBLE 



Figure 4. Profiles of Organizational Conditions and Progressive Instruction 





Note: All of the above scales are standardized to have a mean of 0 and standard deviation of 1 

across 40 states in the 1992 NAEP TSA. 
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Summary and Conclusion 



Content-driven systemic school reform emerged in the past several years 
as a major policy alternative in the United States. This is in part due to the limits 
of current education policy to bring about instructional change at the classroom 
level. To address these deficiencies in educational policy, state policymakers 
proposed reform policies intended to significantly upgrade the quality of the 
curriculum and instruction delivered to all children. Major reforms include the 
establishment of state curriculum frameworks, the development of student 
assessments and the adoption of new textbooks tied to the curriculum 
frameworks. Although many states have recently incorporated the NCTM 
standards into their curriculum frameworks and tests, their classroom-level 
effects remain to be seen. Because successful implementation of content-driven 
reform depends on the linkages between state-level policymaking and school- 
level policy implementation, the central question is how state policies have 
changed the ways in which classroom activities are organized and managed by 
teachers. 

In light of these concerns, this study attempted to investigate the 
multilevel linkages of state education reform to instructional practices in 
mathematics. Four stages of empirical analyses were conducted with a 
combination of the following survey data: the 1992 CCSSO state survey, the 1990 
and 1992 NAEP TSA 8th grade school survey, and the 1990-91 SASS public school 
administrator survey. First, this study examined the linkages among three major 
policy instruments of content-driven reform, that is, curriculum frameworks, 
student assessments, and textbooks. At the state level, the development of state 
tests is more closely linked to state curriculum frameworks than is the adoption 
of textbooks. Likewise, at the school level, the perceived impact of state 
curriculum mandate on instruction is more strongly associated with the 
perceived impact of state testing mandate on instruction than with the perceived 
impact of new textbooks on instruction. Further, this study found that the 
advancement of state curriculum and testing policies is related to modest 
instructional change. These patterns suggest that current state curriculum 
frameworks need to be aligned with instructional materials as well as with 
student assessments. Consequently, the integration of stale policy instruments 
would act to reinforce instructional change at the school level. 
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Second, this study proposed a contingency model of educational policy 
implementation and tested the hypothesis that the effectiveness of state 
education reform depends on a fit between key characteristics of the policies 
(policy goals, policy mechanisms and policy benefit/cost) and the 
institutional /organizational context of the state education system (professional 
values, the distribution of power and school conditions). At the state level, 
principals' greater emphasis on academic excellence turned out to promote the 
implementation of state testing and public reporting policies. At the same time, 
the state department of education's greater curricular influence turned out to 
increase the impact of state curriculum and testing policies on instruction but 
decrease the impact of national reports. At the school level, the socioeconomic 
status of schools is negatively related to school responses to top-down policies for 
instructional change, whereas the level of school improvement activities is 
positively related to instructional change due to external forces. These findings 
imply that policy implementation is a dynamic organizational process of 
interactions between policy characteristics and the institutional setting in a 
multi-layered school system. Thus, systemic school reform needs to formulate 
desired connections between organizational levels in normative, structural and 
functional arenas. 

Third, this study explored the effects of content-driven state policies on 
instructional practices through a multilevel analysis. Principals' reports about 
instructional change due to state policies were used as policy implementation 
variables, which in turn are related to an instructional practice variable created 
from teachers' responses. Insofar as instructional change is intended to ensure 
every student a reasonable opportunity to learn what is recommended by the 
NCTM standards, the findings suggest that the impacts of state curriculum and 
testing mandates on pedagogical practices between 1990 and 1992 are not 
substantial. This study also found the negative effect of district/ school tests and 
the positive effect of new textbooks on instructional practices during the same 
period, whereas the effects of public reporting and national reports turned out to 
be insignificant. Although these patterns indicate that instructional change is 
more likely to be induced at the district or school levels than at the national or 
state levels, the period of this study in examining instructional change may not 
be long enough to evaluate the effects of recent state content-driven policies 
since the NCTM standards for curriculum and for teaching were released in 1989 
and still have been adopted by states and introduced into schools gradually and 
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naturally. Thus, a subsequent follow-up study is needed to explore longer-term 
policy effects. 

Fourth, this study compared two states, California and Minnesota, to 
enrich the overall findings. Differences between the two states were detected in 
state-level curricular decision-making and school-level instructional change. 
Minnesota took a short-term incremental approach to curriculum reform and 
produced outcome-based documents that recommend a broader range of content 
at all grade levels without specific reference to pedagogical details. In contrast, 
California took a long-term comprehensive approach to curriculum reform and 
established a state framework that includes demanding content standards, 
instructional examples, activities, materials, and technology, and assessment 
exercises for implementing content in schools. While results-based initiatives 
assume that state or national standards will lead local educators to create 
curricular that will best meet their particular needs, content-driven reforms 
assume that once xnaterials are selected and matching assessments created, local 
educators will learn and implement the new curriculum. While the two states' 
ultimate policy success remains to be seen, the findings of this case study show 
that, given their different institutional and organizational contexts, California's 
content-driven reforms have led to more successful instructional change than 
did Minnesota's outcome-based reforms. 
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Appendix 

School-level Predictors 

The following variables are constructed from the 1992 NAEP TSA 8th grade 
teacher and administrator survey data. 

1. The Social and Racial Composition of Schools 

Socioeconomic Status: A factor composite of students' reports about parents' 
education and availability of reading materials at home, and school median 
income (school-level average of factor made from PARED, HOMEEN2, 
MEDINC). Student-level factor loadings are as follows: PARED, .77; 
HOMEEN2, .68; MEDINC, .63. Factor has an eigenvalue of 1.46 and explains 49 
percent of the combined variance. 

Percent White: Percentage of 8th grade white students in school (made from 
students' race variable, RACE). 

2. Teaching and Learning Environinent of the School 

Absence of Problems: A factor composite of principals' reports about absence 
of schoolwide problems ifT the following aspects : student tardiness, 
absenteeism, cutting classes, physical conflicts, drug/alcohol, teacher 
absenteeism, racial and cultural conflicts, and student health (factor made 
from C032401-8). School-level factor loadings are as follows: C032401, .71; 
C032402, .71; C032403, .70; C032404, .74; C032405, .49; C032406, .62; C032407, .61; 
C032408, .64. Factor has an eigenvalue of 3.47 and explains 43 percent of the 
combined variance. 

Communal Climate: A factor composite of teachers' reports about 
positiveness of school climate in the following aspects: teachers' relations 
with administration, teacher morale, student attitudes to academics, teacher 
attitudes to academics, parent support for academics, regard for school 
property, and relations between teachers and students (school-level average of 
factor made from C032501-7). Student-level factor loadings are as follows: 
C032501, .63; C032502, .71; C032503, .74; C032504, .69; C032505, .69; C032506, .67; 
C032507, .75. Factor has an eigenvalue of 3.40 and explains 49 percent of the 
combined variance. 
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3. School Capacities for Bottom-up Change 

Program Activities: A factor composite of principals' reports about school 
improvement activities in the following aspects: involving parents as aides 
in class, encouraging parents to visit classes, having minimum requirement 
for homework, performance-based competition system for teacher, mentoring 
program for teachers, before/ after school remediation program, summer- 
school program, and dropout prevention program (factor made from 
C032207-8, C032301, C032303-6, C032314). School-level factor loadings are as 
follows: C032207, .49; C032408, .56; C032301, .33; C032303, .28; C032304, .45; 
C032305, .57; C032306, ,54; C032314, .50. Factor has an eigenvalue of 1.81 and 
explains 23 percent of the combined variance. 

Professional Training: A factor composite of teachers' reports about their 
training in the following areas: estimation, math problem-solving, use of 
manipulatives, use of calculators, students' math thinking (school-level 
average of factor made from T041701-2, T041708, T041704-5). Student-level 
factor loadings are as follows: T041701, .70; T041702, .67; T041708, .69; T041704, 
.68; T041705, .65. Factor has an eigenvalue of 2.31 and explains 46 percent of 
the combined variance. 

4. Instructional Context of the School 

Teacher Autonomy: A factor composite of teachers' reports about their 
instructional autonomy in the following aspects: selecting instruction 
materials, deciding course content/topics, and deciding sequence of content, 
(school-level average of factor made from T041 101-3). Student-level factor 
loadings are as follows; T041101, .84; T041102, .89; T041103, .82. Factor has an 
eigenvalue of 2.16 and explains 72 percent of the combined variance. 

Ability Grouping: A dummy variable for ability grouping pf 8th-graders in 
math (C034402). 

5. Location of the School 

Urban Location: A dummy variable for the location of schools in urban area 
(SURE AN). The base category for comparison is suburban location. 

Rural Location: A dummy variable for the location of schools in rural area 
(SURBAN). The base category for comparison is suburban location. 
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State-level Predictors 

The following variables are constructed from 1990-91 SASS^ public school 
administrator survey data. 

1. Power Distribution in Curricular Decisions 

State Influence: State average of principals' report about the influence of state 
department of education on curricular decisions (SEACURRC). 

Local Influence: State average of principals' report about the influence of local 
board of education on curricular decisions (BRDCURRC). 

Principal Influence: State average of principals' report about their own 
influence on curricular decisions (PRNCURRC). 

2. Professional Values in Educational Goals 

Basic Skills: logit of percent teachers giving high priorities to basic literacy 
skills (reading, math, writing, speaking) as educational goals (ASCI 15-7). 
Academic Excellence: logit of percent teachers giving high priorities to 
academic excellence as educational goals (ASCI 15-7). 

Personal Growth: logit of percent teachers giving high priorities to personal 
growth (self-esteem, self-knowledge, and so on> as educational goals (ASC115- 

7). 

Work Habits: logit of percent teachers giving high priorities to work habits 
and self-discipline as educational goals (ASC115-7). 

3. 1990 Progressive Instruction 

90 Math Instruction: State average of a Rasch measure of progressive 
instructional practices constructed from the following eight items in the 1990 
NAEP TSA data: emphasis on reasoning /analysis (T031511), emphasis on 
communicating math ideas (T031512), using measurement /geometry 
(T031404), calculators (T031405) and computers (T031406), having students 
work in small groups (T031403), and having students write reports/do 
projects (T031410). 



1 The 1990-1991 Schools and Staffing Survey (SASS) offers researchers the opportunity to study 
teachers' and principals' goals and beliefs on a much larger scale than that afforded by the 
smaller scale or case study research typically available to them. 
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4. Instructional Change Due to External Forces 

State Curriculum: Logit of percent principals who report instructional 
change due to state curriculum mandate (C032603). 

State Testing: Logit of percent principals who report instructional change due 
to state testing mandate (C032604). 

District/School Tests: Logit of percent principals who report instructional 
change due to district /school tests (C032602). 

Public Reporting: Logit of percent principals who report instructional change 
due to public reporting (C032606). 

New Textbooks: Logit of percent principals who report instructional change 
due to adoption of new textbooks (C032601). 

National Reports: Logit of percent principals who report instructional 
change due to national reports (C032607). 
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